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An internet network force feedback system (10) which 
includes a server machine (18). a client machine (14) provided 
with the force feedback device (24). and one or more additional 
client machines (16) each of which may be provided with 
additional force feedback devices (26). The server machine 
IS a computer or processor ninning TCP/IP server software and 
IS connected to the Internet Client machine (14) njns internet 
browser software. The force feedback device (24) has sensors 
and actuators to monitor the input of the user and transmit the 
mpui to the client and receive response from the client and 
provide the force feedback sensations to the user. 
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Method and Annaratiis for ProviHin g Force Feerthack Over a romp utp.r Np.iw»rk- 

Background of the Tnvenfinn 

This invention relates generally to human/computer interfaces, and more particularly to 
human/computer interfaces with force feedback that can operate over a network. 

1 0 The Internet has. of late, become extremely popular. The origins of the Internet date back 

several decades to a U.S. government sponsored miiitaiy/researchAjusiness wide area network 
(WAN) that was designed to remain operational even in the event of the catastrophe, e.g. a major 
eanhquake or a nuclear war. To accomplish this goal, robust protocols and syswms were 
developed which allowed a geographical]/ distributed collection of computer systems to be 
connected as a WAN such that the loss of a particular computer, or group of computers, would 
not preclude the continued communication among the remaining computers. 

While the use of the Internet has been prevalent for many years now. its use has been 
limited by the arcane and difficult commands required to- access the various computers on the 

network. To address this problem, a protocol known as the "World Wide Web" or "WWW" was 
developed to provide an easier and more user-friendly interface for the Internet. With the World 
Wide Web. an entity having a domain name creates a ^'web page ' or "page" which can provide 
information and, to a limited degree, some interactivity. 

A computer user can "browse", i.e. navigate around, the V.T,-W by utilizing a suitable 
web browser and a network gateway (e.g., an Internet Service Provider (ISP)). For example, 
UUNET. America Online, and Global Village all provide Internet access. Currently, the most 
popular web browser, known as the Netscape® Navigator®, is made by Netscape Corporation of 
Mountain View. California. The web browser allows a user to specify or search for a web page 
on the WWW, and then retrieves and displays web pages on the user's computer screen. 

The Intemet is based upon a transmission protocol known as "Transmission Control 
ProtocoWntenet Protocol" (or 'TCP/IP" for short), which sends "packets" of data between a 
host machine, e.g. a server computer on the Intemet, and a client machine, e.g. a user s personal 
computer connected to the Intemet. The WWW is an Internet interface protocol which is 
supported by the same TCP/IP transmission protocol. Intranets are private networks based upon 
Intemet standards, and have become quite common for managing information and 
cotmnunications within an organization. Intranets, since they adhere to Internet standards, can 
often use the same web browser software and web sei^^er software as are used on the Intemet. 

A web page typically includes static images and text. The images and text are specified 
m a "HyperText Mark-up Language" ("HTML") file that is sent from the web server to the client 
machine. This HTML file is parsed by the web browser in order to display the text and images 
on the display of the client machine. Other standardized languages or protocols are also being 
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developed for us. wi,h ,U ,„„™„ wortd Wide Web For exa„,ple, .He V™., Rea,i,v 

Model.". Wu.ge (VR*,U . .ed ,„ provide vi.. vi„„a, 3-0 e„v.ro„l„,. Jlw o e o 

"ZT"' """"^ '"^ - - = 

TT« h-e^e, and .he WWW al«, pe™ .o„„d daia ,„ be iransmiaed over ,he In.e™. 

«^ H<*t.Keis coaed m TCP/IP format can aiso be sent fmm ^r,^ i 

:r,:r:re::e:r"--"^'-"^-^™^-— 

Wlete^smission of visual images (bo.hsuiic and dynamic, (e video) ,ex, a„rf 

oeen well explored In paracular. (he transmission of data over the Intemei ^„.- 
sense of toucb and/or force b„ no, been establtshed. -Fol td^T- ZT'"' 
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The present invention is related to the transmission of information pertaining to a subset of 
the sense of touch, i.e. the transmission of forces to a user over a computer network svstem TTie 
"force feedback" provided by the methods and apparatus of the present invention enhance the 
) sensory experience of the user to provide a richer, more interesting, and more enjoyable experience. 

In a preferred method of the present invention for providing force feedback over a network a 
connection is first established between a server machine and a client machine. TT^e client machine 
(typically a personal computer), has a visu4l display (e.^.. a computer video monitor) and a force 
feedback human/computer interface device. Examples of force feedback human/computer interfaces 
(hereafter "force feedback devices") include force feedback joysticks, mice, trackballs, steering 
wheels, and yokes. In some embodiments of the present invention, the force feedback device 
preferably constrains movement to two degrees of freedom to match a two-dimensional configuration 
of the visual display. Three or more degrees of freedom of movement can be provided in other 
embodiments. Next, the client machine receives from the server machine both screen display 
mformatron and force feedback infomtation that is related to die screen display information 
Preferably, the screen display information and force feedback information are encoded in an IfTML 
web page file. Files and data in od^er protocols or languages can also be used, such as VRML 
Next, the c!:=nt machine displays on the monitor an image generated from the scr^n display 
mformatron. The force feedback device provides a pointer event and/or a button event to the client 
with respect to the screen image, and a force feedback signal is computed based upon the detected 
events and the force feedback information stored on the client machine. Finally, a force feedback is 
provided to a user by the force feedback device based upon the force feedback signal. 

Preferably, the force feedback device is provided with a local microprocessor which 
communicates with the client machine. TTie force feedback device further includes sensors and 
actuators coupled to the local microprocessor such that the force feedback signal can take the form of 
a relatively high-level force feedback command. The local microprocessor parses the force feedback 
command to control the actuators of the human/computer interface in a control loop with the sensors. 

Another preferred method of the present iavenUon for providing force feedback over a 
network establishes a connection between a first computer and a second computer over a network 
The first computer includes a computer input device which develops a first computer input and a 
second computer includes a visual display and a force feedback device for providing a second 
computer input. The computer input device of the first computer may or may not also be a force 
feedback device. A screen image is displayed on the monitor of the second computer that is 
associated with stored force feedback information. A computer input is received from the first 
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desired URL 1, """^ """"""^ *«• is team a 

deve opine a force feedharl. cim,,! fur rcrcrence and the parameters; (c) 

e«pc.e„ce Jeree..:;::;e„e.!:r'"" ' " 

pe peer basis, o, f,™„ a server lo one or mc, peeni i„ a server-lo-peer basis. 

co.p..r;srrMirr r? : ""^'^ "^"^ -"-"^ = --^'^ 

-ver maeh™ e„„,^ed ,„ .he new^ „r i.,,^,^ can te passed be,ween ,„o or 
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more client machines on the network. When force information is downloaded from a server, the 
client machine parses and interprets the force infomiation and directly controls its interaction with the 
force feedback device on an essentially real-time basis. In contrast, peer-to-peer or server-to-peer 
direct interaction over the network may be subject to some transmission ("latency") delays, but 
permits remote interactivity with a client's force feedback device. 

These and other advantages of the present invention will become apparent upon reading the 
following detailed descriptions and studying the various figures of the drawings. 



* 

I 
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F.6. 2 .s a blocMagram of a diem machine i„ u,e p^e„, ,nve„,i„„; 

Fig. 4b is a cross-sectionai view taken along line 4b-4b of F.g. 4a: 
~e — - ^ fo.e fee.a. .vice in - 

Fig. 5a;'''' " ' °' ' ^« feedback device of 

of Fig. Z ' °' ' ^'^'""^ °f ^he force feedback device 

Fig. 6 is a block diagram of a wide area network (WAN) based upon Internet TCP/TP 
protocol and supporting World Wide Web fWWW^HTK/n . a upon Internet TCP/IP 

invenUon; ^^^^^^ance with the present 

invention!'' ^ "''^^"^^ --^-^ -i^'' the present 

Fig. 7b is an example of an HTML file of the present invention sent from a web server 
machme to a client machine; 

Fig. 8 is a flow-diagram of the "Pan^e and Inteq^t^t HTML Component" step of Fig. 7a; 
Fig. 9 is a flow-diagram of the "Plug-in Interpret .IFF File" step of Fig. 8; 

Fig. 9a is an illustration of an image displayed on a visual displav of a client computer as 
generated from a downloaded HTML web page file; and ' 
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Fig. 10 is a flow-diagram of a process for monitoring the "pointer state" of the force feedback 
device of the present invention. 
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m F.g. , a network system 10 includes a w.de area network (WAN) such as the Internet 1 ^ 
mach.„e M. a second client tnachine ,6. and a web server machine 18, are coupled to the Intl 
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As noted prev.ousiy, both the Internet 12 and Intranets opetate using the same TCP/IP 
protocols, ^.s allows Intranets to use sit... or the same server mach.ne softwrid cZ 
machme software as a« used in Interne. ,2 Applications. Therefore, it wil. be appar n ' I 

fonns Of network systems that are compatible with the processes and apparatus disclosed herein. 

n^edia 2^' °' '"^"'^'^ ^-mission 

media 22. Tl,ese nodes are typically routers, switches, and other mtelligent data transmission 

.nstances. the nodes 20 compnse an Internet service provider (ISP) 20a which allows a client 
n.ach.netoaccess the "backbone" ofthe Internet. Alternatively, client machines and web server I 
be.coupled directly into the backbone of the Intemet. 

As noted previously, the pt^sent invention is directed to the implementation of force feedback 

force feedback, force feedback human/computer interfaces (hereafter ' W feedback devices ') 24 
nd 6 can be provided as pan of the client machines 14 and 16. respectively. Tl,e client machines 

"vir, \ " T ' ""'"^^ ^"^ ^« ^-^-h - o- example of a 

vist^al display )• respectively, which can display images 1 1 and 12. respectively. P,.fer.bly fo«:es 

each h "^'^ '"•^"^^^ °f '''^ '° ^ es," and 

ntntn" T" '^"^"^ ^ ^ one embodiment of the present 

■nvention, a cliem machine, such as client machine ,4 or 16. sends a request for a "web page" 
residing on. for example, web server machine .8. This is accomplished by the client macl 
sending a connection request and a URL which specifies the address of the web page to the web 
^ZoZ'^Z a web page 32 wntten m HTML fonnat 

dtk o H ' " ' "^"^'^ J^AM. hard 

disk, or a combination of the two) of the client machine. In this embodiment of invention the 
.mage on the video display of the client machine is generated from the HTML web page file cadied 
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5 on the client machine, and force feedback is provided to a user through the force feedbacic device as 
he manipulates a user manipulable object of the force feedback device. 

In another aspect of the present invention, a first client machine, such as'.client machine 14, 
and a second client machine, such as client machine 16. directly communicate force feedback 
command., to each other ir. standard TCP/IP protocol over the Internet 12. More particularly, client 

1 0 machine 1 4 can send force feedback and other information to the URL of the client machine 1 6. and 
the client machine 16 can send force feedback and other information in standard TCP/IP packets to 
the URL of the client machine 14. Inthis way. users of client machine 14 and client machine i6can 
interact physically over the Internet 12. Of course, a server machine 18 can likewise directly 
communicate force feedback commands to a client machine 12 or 14, or all three machines can 

15 interact. / 

In Fig. 2. a "personal" computer 34 architecmre that can be used for client machine 14 or 
client machine 16 is shown in block diagram form. It should be noted that a variety of machine 
architectures can be used to access the Internet 12. i.e. can be used as "network access computers." 
The particular architecture shown for the computer 34 is a typical personal or "PC" computer 
architecture, such as that used with IBM compatible personal computers. Web server machines can 
also have similar architectures, but are often more powerful computers known as "workstations" that 
operate under some variant of the UNIX« operating system. TTie Internet senrice providers 20a are 
likewise often UNIX-based computers or powerful personal computers running Windows NT*. The 
nodes 20 are most commonly routers built by Cisco Systems of San Jose. California. Client 
machine 14 or 16 can also take other forms, such as a television including or connected to a 
microrprocessor for Internet access. Force feedback devices used with such client machines can be 
appropriate for the particular embodiment, e.g., a TVremote control used for internet browsing on 
the abovementioned television can include force feedback functionality. 

The personal computer system 34 includes a microprocessor 36 clocked by a system clock 
CLK and which is coupled to a high speed or memory bus 38 and to a lower speed or L'O bus 40. 
The system RAM 42 and ROM 44 are typically coupled to the high speed memory bus. while various 
peripherals, such as the video display, hard disk drive, Internet interface (often either a modem or an 
Ethernet connection), and force feedback device, are typically coupled to the slower I/O bus. The 
microprocessor executes programs stored in the various memories (RAM. ROM, hard disk, etc.) of 
the personal computer 34 to control, for example, the image display on the video display and the 
forces provided by the force feedback device. The manufacture and use of personal computers, such 
as personal computer 34. are well-known to those skilled in the ait. 

In Fig. 3. a client machine 46 in accordance with the present invention includes a personal 
computer system 48 and a force feedback human/computer interface or "force feedback device" 50. 
A user 52 can receive visual information 54 and auditory information 56 from the personal computer 
48 and can manipulate the force feedback device 50 as indicated at 58a and 58b to provide input, 
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P.ys.ca, '.eer slslr; ' '"^^^^ ^^^^^^ ^0 . .prese. 

o.e.co.po„e„.or.ep_a.co.pJ3...4S.^ 

dev,ce 66 are typically coupled to the I/O bus 40(„o, Shown in .his fi^^^^^^^ 

In this preferred embodiment, the force feedback device 50 orefer^h. ■ . . 
microprocessor 68. a local clock 70 on.i. J, , Preferably includes a local 

. - locai Clock /O, optionafl local memory 71 for the Inrai rr,;^ 
' sensor, interface 72 sensors 74 « .,c • . nucroprocessor 68. a 

^ i-c, sensors /4, a user manipulatable obieci if, "^.h^,- ■ • *" 
actuator interface 80. a safety switch 82 .nH I "'"^ ^''J^'^' ^6. other mput mtetface 78, an 

be through a strial p„„ couplin. 88 to ,h, , ^"^ ™" ""™»""<:»tio„ coupling ea„ 

BM PC/AT s,a„dar* w,l, ,„c,I a" X " T ' '° 
tw<.»a,oo™,u„.a,,o„ JlTlCo^-::^'"'^ M „o«d, s^a, po„ win po™, 
over the game pon coupling whichTr ™:™prooe«or 38. and is prefe*l= 

to .ho miLpr^oT' """"-"^O" f"»" *e local p,oc^r 68 

system 48 a3^1 ~' ' ~" "^""^ "'^•e' 

able ,0 provide U,is func,i„„ J"""""^ """I' > 8"™ P«t »nn=cuo„ alone „„, ^ 
only a gan,e iZ^J^J^T.'^T'^''^''''' ™-Proeesso, 68 even if 

nt^c^pL.: .rc:™ca^^rr :ra,":r ^ ^ 

:»enal Bus (USB) of a personal computer, although an RS c^rioi k . 

parallel bus. an ethen,et bus. or other type o ilterf ' " ""^ ' 

or other types of mterfaces or communication links can also be used. 

icon tonove'in t ™,e d : LTr' *= «bi- » ca«. a ••poin.e," 

76 and co™™,nica,e3 .h. mov«ll"o * ""i^" 
Tlte local micopraces^rr^ "-"rop^cesso, 68 through d,e „„»r imerface 72. 

rnic^p.^ j^seit ~r36r rr " d*"™ " ^ " - 

P cssor 36 to create a corresponding movement of the pointer 
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icon on the image displayed upon d,e visual display 64. In some embodiments, die sensors 74 cm 
communicate directly to microprocessor 36 without the use of locaJ microprocessor 68 The user can 
also create other input, such as a "button click," through the other input 78 which are communicated 
to the microprocessor 36 by the local microprocessor 68 or directly, e.g.. using a^ame port. 

If the pointer icon on the display device 64 is at a position (or time) that correlates to a desired 
force feedback to the user 52, the microprocessor 36 sends a force feedback command to the local 
microprocessor 68 over the serial port connection 88. The local microprocessor 68 parses this force 
feedback command and sends signals to the actuator interface 80 which causes the actuator 84 to 
create forces F on object 76. which a^ experienced by the user 52 as indicated at 60 -Rie safety 

as a "deadman switch", blocks the signal from the actuator interface 
15 80 If. for example, the user 52 is no longer grasping the object 76. In this wav, the user 52 can 
mteract w,th the client machine 46 in a visual, auditory, and tactile fashion. ' 

In Fig. 4a. a force feedback device 50a is provided with a user manipulatabie object 76a 
Which, .n this instance, includes a shaft 90 and a ball-grip (or joystick) 92. The force feedback 
device 50a also includes a pair of linear voice coil actuators ("voice coils") 94 and 96 that can serve 
both as sensors and actuators. Altertiatively. the voice coils can be used only as actuators and 
sepa^te sensors (not shown) can be used. The voice coil 94 is coupled to the shaft 90 of object 76a 
by a first link 98 and a second link 100. Link 98 is coupled to link 100 with a pivot 102. and a link 
100 .s coupled to the shaft 90 by a pivot 104. Similarly, voice coil 96 is coupled to the shaft 90 of 
the Object 76a by a first unk 106 and a second link 108. The fi,.t link 10. is coupled to second link 
108 by a pivot 1 10, and the link 108 is coupled to the shaft 90 of the object 76a by the pivot 104. 

•nie link 98 can move in and out of a housing 1 12 as indicated by arrow 1 14. and link 106 " 
can move in and out of a housing 1 16 of voice coil 96 as indicated by the arrow , 18. The pivots 
102. 104. and 1 10 allow the object 76a to move within the constraints of an x-y plane, but does not 
permit movement in a z direction orthogonal to the x-y plane. Tlierefore. the fotre feedback device is 
a two degree (2D) of freedom device. That is, die user manipulatabie object 76a can move with a 
frst degree of freedom in a x direction, and in a second degn^e of freedom in the y direction A 2D 
force feedback device 50a is considered preferable in the present invention since it co^elates well to 
tne two-dimensional screen of a monitor of a client machine. 

In Fig. 4b. a voice coil 94 is shown in a cross sectional view taken along line 4b-4b of Fig 
4a. The ous.ng 1 12 includes a centra core 120 and a number of elongated magnets 122. An 
annature 124 includes a hollow, cylindrical member having inner surface 126 which slidingly 
engages the core 120. Wrapped around the armature 124 are coils 128. TTie coils are electric^ly 
coupled to actuator and/or sensor interfaces. A plate 130 is attached to the end of the armature 124 

TZ Tk """^ -"^^^ - ^ - i-'iicated 

at 1 14. Other vo.cecoil configurations can also be used, such as differently shaped cores, different 
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coil layouts. «c. In addi.ion. „U,c, .yp,. of acm,o. can b. used i„.,ead of „, addmon ,o vo.e 
co„s ,„ fo„. feedback device 50. such as DC n^on. b„*es, pneumatic acualo.. e,c. 

f„.h IT^' ^^T o' ' '""l-k *»ic= 50b is illustrated. The .o™= 

feedback devce 50b has p„i„„ of simiW,, wid, ^ ,^ f.„,bact dev,c; 50a 
reference nun^als refe,.„g ,„ ,ike ele„»„.. v. ,o,ee feedback device 50b Mudes *e „^ 
man,pu .,ab le objec, 76a. d,e sbaf, 90. d,e ball knob 92. and d,e voice coils ,4 and 96. Holvr 

1 : , r ^ -^^ "«">^-. mI' 

pan,cu,arly. ,be hnks 98 and 100 of force fe^back device 50a have been ,eplac«. bv a nZ 

.06 a^ ,08 of .he force feedback device 50. have l.e„ replaced by a conneco, .en,ber 136 and 
15 fl«,ble„„en,t^,38(al»collec,ivelyco™p/isi„gane,„re.e™be,,, T„econ„ec,or I32isri,ir, 

end of ,he nex,ble „,e„ber ,34. Theo,herend of ihe flexible „»n,ber ,34 is anached ,„ a ba.se ,40 
wh,ch. ,„ ,„™ ,s rigid,y ,„rted ,0 d,e shaf, 90 of *. objee, 76a. SimilaHy. ,he connecror 136 is 
anached „ a p,a,e of an a^a^re of voice coil 96 a. one of its ends, and is a«*cd ,o ,he nexiWe 

ZZIT '•'''^'■'^-''^''^^^^^^^ 

■;;;'=""i*'"«"'««'13<«»<il38se,veU«aanKfunc.io^ 
d=v,ce 50a descnbed previously. As d,e objee, 76a is moved back and f»,h along an x-y p^re le 
ne.b e .ember 1 34 can „ove in and on. of 0,e voice eoi, housings 94 and 96. respecively and can 
25 bend .0 acco^nodare angnl. n»vemcn, with .espee, ,o *e x and y axis, nis pL. Z 
connecors ,32and 1 36 <o move back and fo„h w,*in d« voice coils 94 and 96. respecively 

In Fig. 5b. an aUeraative user manipulauble objee. 76a Bkes form of a slylus 142 which 
can engage an aperture ,44 in an alterative base , 40'. The .lten,a.ive base 140' can be coupled to 
30 ITT '^«°f'^"-l-'™->fF*5». Al«=r„advely.«,e.ip of stylus ,42 

ioTnttZlr ' « 

In Fig. 5c, another altemadve base I40" is pt^ided wid, «, enlarged apemne I44' which 

nl I'T?" °' ' °' "">•■ -«r 

35 tZ f " 134 and ,38 of 

35 the first feedback device 50b of Fig. 5a. 

The embodiments of figures 5b and 5c illustrate two of a range of equivalent user 
n.an.pu a.bie objects suitable for the present invention. It should be apparent to those skilled in the 

force feedback devices, such as the force feedback device 50a illustrated in Fig. 4a. 
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As noted previously, a preferred embodiment of the present invention provides a user 
man.pulauble object that has two degrees of freedom. Other user manipulatable objects having one 
degree of freedom or three or more degrees of freedom are also within the scope of the present 
invention. For example, one embodiment of the present invention provides 6^ly one degree of 
freedom. Other force feedback devices of the present invention include mice, joysticks, joypads a 
steermg wheel, and yolks having two or more degrees of freedom. 

In Fig. 6. a conceptual representation of the network system 10 with force feedback includes 
a server machine 1 8. a client machine 14 provided with a force feedback device 24. and one or more 
addutonaJ client machines 1 6. each of which may be provided with additional force feedback devices 

the TPPr' ''"f """" ' ""''""^ —Pie. 
the TCP/IP server software and is which is connected to the Internet. The client mach.ne 14 includes 
a computer or "processor" running Internet browser software and force feedback dnver software 
The processor of the client machine is connected to the Internet and to the force feedback device 24^ 
The force feedback device 24 has sensors and actuato.. so that it can track movement of the user 
man.pu atable object, monitor for button presses and/or other ancillary input devices, and p«,vide 
output force feedback sensations. The force feedback device 24 sends object tracking information to 
the chent mach.ne. and receives force feedback commands from the client machine 14 n,e 
"add.t,onal client", such as client machine 16, also includes computer, or "processors" runnini 
Internet browser software and force feedback driver software. The processors of these additional 
clients are also connected to the Internet and are connected to force feedback devices associated with 
ZD that client. 

As noted in Fig. 6. a client machine 1 4 can send a data request to the server machine 18 and 
m re^. recetve an HTML web page file including a special file of the present invention known as' ' 
an IFF f.le. As will be appreaated by those skilled in the an. the server must also have a modified 

.0 Tp m'^omc ' "^'^ ^ '>'P^ -^h- -^•f-'^ would be 

30 SRM-CONF or other .CONF ftle. TT.e client machine ,4 then .sends force feedback commands tc 
^ force feedback device 24 and receives tracking and button data f«>m the force feedback device 24 
Ghent machine 16 can likewise send a data request to the server machine 18 and receive an Him 
file w,th one or more IFF files. TT^e client machine 16 can ti,en interact with the force feedback 

35 dtT '° '''' ^"^ ---g ^-'^"g -d button 

35 data from the force feedback device 26. 

In addition to communicating with the server machine, the client machines can communicate 
ditectly w.th each other over the Internet using an Internet communication protocol. For example 
chent machme 14 can communicate with client machine 16 through a TCP/IP connection This is 
accomplished making the URL of the client machine .6 known to the client machine 14 " and vice 
versa. In ^,s fashion, din=ct communication between client machines can be accomplished without 
.nvolvng the server machine ,8. These connections can send force feedback information and other 



a 
to 



40 
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force feedback ,„f„n.a„o„ o«r a TCP/IP .„,»„,« connecta ,o .he diem „acl,i« ,4. which w.ll 
he„ ge„era,e a fo„e feedback command ,o .he fo,c= feedback device 24. When user reads ■„ Z 
orce feedback a. force feedb^k device 24, ,his i„fo,™a,i„„ can b. sen, fron, Z 
d.en.m.ch,ne 16 to provide force feedback ■„,,„ „ser on force feedback device 26. 

' nrachin P">'«^ ™«-"» on a ci.em 
mach,ne, such as Cen, n«chi„e ,4 or dien, machine 16. TOs ^ ,^ I 

of force feedback commands. A p,efe„ed browser software is Netscape NVvi„,„^ 7 
av^abie from Ne.cape Co,ora,io„ of Moun^n View. C.ifo™ia. "Ta 
P„e«e.io„„f rhe „d, browser sof/ware. where .his p^pne,^, e„e Jon was de'c^d 
. by the Applicant Of the present application. .. = eiopca 

TZ ^rT J " ' URX- is *e URL 

col^ J ""'V ' " ^' 

^mmands. A„en«„,ve„, rhe d«>ired web page can reside „„ a„„,her server or resource and be 

rerr, vcd by serve, mad^e ,8. In response ,o .he connection reques. of s,ep ,50. ,he server - 
rnac ,„. ,8 sends MML ffle representing ^ web page over ^ fntemct to e ,cc ived h i 
tZTr "^'■•"■•-"'^''"^of "components- which a. typicaiiv commL, 
command ftagments. tnstructions. and d«. whid, penttit the display of the web page and other web 

o f„e , eof ,, a step ,56 detects that fact and the process is comp,eu=d a. ,58. Otherwise, the 
HTML component ,s pa,s«, and interp^ted a, a step , 60 and process conttol is tetumed at step 154. 

processtng, the HTML components even before the entite HTML ,s teceived Z the dU, 
madttne. Aitema.ively. dte entire HTML file can be ..ceived bef^c the pt^ing begins. 

"S. 'b. an example of an HTML web page 32, sent ftom a web s«ver machine 18 to a 
cl.en,m«:h,nc(suchasclientmachinel4orI6)overd«l„temct 12. is shown. TTte WML fik 3^ 
jndudes a number of ••components" which are parsed and intetpreted as p^viously described. An' 
mm me beg.ns w,th a <HTMl>command ,62 to indicate the Stan of the HIML file, and a 
n^^V™"" '° '^^ "™'- Uen. an arbitrary 

2 vOeo d.sp,a, of the dien, mad,ine. A <CENTER> command ,68 wi„ cause a centering of 

Next, an <EMBED ...> command 170 of the p„=se„t i„v«„i„„ dete a force button object that wt„ 
be^sp^yed on the Cent machine. Since the <CENTER> command 168 was gi«=„ just prior to dte 
<EMBED ...> command, this "force buuon" wi,l be centered in th. displayed browser window 



10 



20 



30 



35 



wo 98/06024 

PCT/US97/12848 

Other force objects besides bmton objects can also be defined and displayed, such as links text 
sl.ders. game objects (balls, paddles, etc.). avatars, windows, icons, menu bars, drop-down menus " 

or other objects. 

In a first line 172 of the <EMBED ...> command, the force button object is defined by a 
IFF extension file, namely "FORCEBUTTON.IFF." .Next, in a line 174. the size of the button .s 
.nd.cated to be 100 pixels by 100 pixels. In a line 176. the initial state of the button is indicated to be 
•up" (..e.. unselected). and a line 178 defines the force effect to be "vibration." A number of 
parameters 180 defining the character and nature of the vibration are also provided (start time length 

Zv7e.^^'"T- ^" ' ""^ ' '''''' « 

MOUSEWTTHIN wuh its associated parameters, and by the function "BUTTONSTATC " The 
function MOUSEWTTHIN determines wheWr the pointer ,con. the position of which is controlled 
by the force feedback device, is within the specified boundaries defining a region of th^ force button 
This region can be specified by the parameters and, for example, can be defined as the exact 
disp ayed area of the button, or can be defined as a sub-region within the button that is smaller than 
the displayed size of the button. TT^e function BUTTONSTATC detennincs whether a button or 
swuch of the force feedback device is in the desired state to trigger the force object event (e g a 

^nnrril'." '''' ' '''' "^""^"''"^ - ^P-^^ed'as 

LOUIS.GIF. and the text associated with the button is defined as "Hi, I'm Louis" in a line 186 

The font of the text .s given as "Helvetica" in a line 188. Other force effects, trigger, and parameters 
can also be associated with the force object. For example, a force (such as a vibration) can be 
tnggered if the pointing icon is moved a predetemiined velocity or within a predefined ranp^ 
velociti« within the force object. Or. a trajectory of the pointing icon on a fore object can trigger^a 
force, like a circle gesture. 

The <EMBED ...> command is an existing ftmctionality of HTML. It essentially embeds 
function calls which are handled by the web browser. If the suffix of the specified file is a known 
standard suffix type, the call is executed directly by the web browser. If. however, the suffix ( IFF 
m this instance) is not a standard feature of the web browser, the browser will first look for a "plug- 
m to .mplement this feature and. if a suitable plug-in is not found, it will look for application 

programs .mplementing this feature. In the preferred cmbodimem of the present invention,apIug-in 
.reading a reference to a Dynamically Linked Library (DLL) is provided to give ftmctionality to the 
•IFF suffix. The DLL can be provided local to the client machine or on another linked ) 



I resource. 



With continuing inference to Fig. 7b, the centering command is lemiinated at line 190 with 

If'-r ™ ^"'""^^ ^ P™^'*'^^'. bodv 

of the HTML file .s terminated by the </BODY> command 194. TTie end of the Him file i's 

40 HTrfilir ^'"^ ''' ''"^ Of ^ 
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The present invention also provides for programmability of the embedded force feedback 
object. An example of th,sprogrammabiIity is shown at 198. This optional programmable command 
can be msened into ,he EMBED command 170 and can mclude. for example, an iterative loop In 
ime 200. a "FOR" command initializes a counter i to 0. indicates that the counter 1 i^- incremented by 
one per each pass through the loop, and it indicates that the loop should be completed five times / e 
while ,<5. The body of the loop includes a command line 202 which indicates that a force feedback 
■vibrate" w.th associated parameters should be evoked, and a line 204 indicates that a 5 second wait 
should be provided after the vibration has occurred. This step will repeat five times ie the 
command 198 will cause f.ve vibration sequences separated by four 5 second pauses, and followed 
by a final 5 second pause. By providing programmability to the force feedback object force 
feedback effects based upon past events and u^n a complex interaction of factors can be provided. 

In Fig. 8. the "Parse And Interpret . mm Component" or simply "Proce'ss mm 
Component" step 160 of Fig. 7a is illustrated in greater detail. In E.g. 8. process 1 60 begtns at 206 
and. m a step 208, it is detemiined whether there is an embedded "tag" for a force object e . a tag 
havmg an .IFF reference. An example of the embedded tag is shown at the EMBED command 170 
of Ftg. 7b. If there ts such a tag, step 210 uses the plug-in software of the present invention to 
mterpret the .IFF file, and the process is completed at 212. Otherwise, another tvpe of mm 
command has been encountered, and the standard web browser parser and inten^reter'processes this 
HTML component m astep 214, after which the process is completed at 212. 

In Fig. 9. the step 210 Plug-In Interprets .IFF File" of Fig. 8 is described in greater detail 
Process 210 begins at 216, and in a step 218, a "framework" is created for the force object TTk 
framework provides a particular set of generic features to implement the specified force object, and 
preferably mcludes no specific parameters or functions for the force object. Next in a step ^''O the 
name/value pairs are parsed and, in a step 222. the force object ,s built upon this framework based 
upon the name/value pairs. A name/value pair includes the name of a component and its associated 
parameters. For example, one name might be "BUTTONSTATE" and its value (or parameter) might 
be "UP" (or "UNSELECTED"). The process 2 10 is completed at 224. 

In Fig. 9a. an image 226 to be displayed on a screen of a video monitor or other visual 
display is illustrated. Moie specifically, image 226 can be generated by the popular Netscape 
Navigator web browser. The image 226 includes a window 228 including a header poaion 230 and 
a body portion 232. The header portion 230 includes a number of navigation buttons 234 and special 
purpose buttons 236 for purposes well-known to those familiar with the Netscape Navigator web 
browser. In addition, the header portion 230 displays the URL of the currently displayed web page 
at 238. In this instance, the URL is "http://www.immerse.com/demo." The images displaved within 
the body ponion 232 of the window 228 are created by the aforementioned processing of the mm 
file by the web browser. 
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The area within the body ponion 232 has been provided with a number of regions and 
buttons 10 illustrate some of the concepts of the present invention. The force feedback device 
controls the position of a pointer icon 240 which can be caused to interact with the various regions 
and buttons. As an example, when the force feedback device is manipulated by the user to cause the 
pointer icon 240 to move within a "texture" region 242. force feedback commands can be created for 
the force feedback device to provide a desired "texmre" to the force feedback device. For example 
the texture can feel "rough" to the user by causing the force feedback device to place forces on the 
user manipulatable object that emulate a rough or bumpy surface. In a region 244 a "viscosity- 
force feedback can be provided. With this form of force feedback, as the pointer icon is moved 
through f.eld 244, a viscous "drag" force is emulated on the user manipulatable object In a region 
246. menial forces can be felt. Therefore. ^ pointer icon being moved through an "inenia" region 
would requ.re relatively little or no force to ''move in a straight line, but would r^juite greater forces 
to accelerate in a new direction or to be stopped. The inertial force sensations can be Ipplied to the 
user manipulatable object and felt by the user. In a "keep ou," region 248, the pointer image is 
prevented from entering the region. This is accomplished by creating a repulsive force on the user 
manipulatable object using a force feedback command to the force feedback device which prevents or 
mhibits the user from moving the user manipulatable object in a direction of the region 248 when the 
pointer icon 240 contacts the periphery of the region 248. In contrast, a "snap-in" reeion 250 will 
pull a pointer icon 240 to a center 252 whenever the pointer icon engages the periphery of the snap-in 
region 250 and apply a corresponding attractive force on the user manipulatable object. A "spring- 
region 243 emulates a spring function such that a pointer icon moving into the spring region 
"compresses" a spring, which exerts a spring force on the user manipulatable object which opposes 
the movement of the pointer icon. A region 256 is a "Force To Left" region where the pointer icon 
withm the region 256 is forced to the left side of the region and the user manipulatable object is . 
forced in a corresponding direction as if influenced by some invisible magnetic force or gravitational 
force. A region 258 illustrates that regions can be of any size or shape and that within a region 
different force effects can be developed. In this example, within region 258 there is a texture core 
260 surrounded by a vibration ring 262. Therefore, as the pointer icon 240 moves into the region 
258. the user first experiences vibration from the ring 262, and then experiences a texture as the 
pointer icon moves within the core 260. 

The exemplary force feedback web page of Fig. 9a is also provided widi several force 
feedback buttons. In a first button 264. the placement of the pointer icon 240 over the button and the 
pressmg of a button (i.e., a switch) on the force feedback device to create a "button click", "button 
down", or simply a "button event" input, will then cause a "buzz" command to be sent to the force 
feedback device. The buzz command would, for example, cause a vibration force on the user 
manipulatable object. Similarly, the selection of the "jolt" button 266 will cause a jolting force (e g 
a short-duration pulse of force) to be provided at the force feedback device, and the pressing of the 
"detent" button 268 will cause a "detent" to be created for the force feedback device. Bv "detent" it is 
meant that the user manipulatable object will be conirellcd by the force feedback actuators such that it 
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' itr;,"rr.::ir ■ - — - - - 

It should be nc^ed tha, ihe foicc feedtack drive, (browser pr„..i„ „, DLLl h k 
ability K inKor, wid, JAVA eode. Inlhi,' embodiment ih. I I" " ^"-> I'-"' ■!« 
.5 co«ds using „e brewer. m„.„me ^A^T^^r mC"'" """" 

••.-.iebpe.o™d,nam,c.ode.su.;~rrerr~^ 

. rrzt::tu„rie:r:— 

Furthermore, the force feedback driver (browser plug-in or DLI h u u-.- 
i«.e^^M.^onspr„.idedi„o.ber,»,u,esbesi^^^^ 

..T "'^ *"^"="'" '">"< 'I""' »n,puters^ Force feedback commands and pamneters 
30 Chen, comp„„„y^ ^ ^ ^ ^ ^ J-ed^V = 

cw»i in Ihe VRML dau fa a vinual environment so that when the user moves into a vimial wall 
obst^ction fo^ i, <,„ ^ user-manipulatable object. Or. „b«i the usercanier. ili 

35 ' 7,7 t'™^ f-' » 'i-n-a.^ weigh, of d. vimial'^ „t 

u^ * C ^'"1 « l^-ides d« 

r^rtoi^cZcr " °' ~ " - - 

alter., '^"'«<' of seveml piefened embodiments diete ,» 

a^so be no«d *a, d»,e „e may al«madve ways of implemenung bo,h d« process and appamus of 
the presen, ,nve„,io„. „ is dierefo^ .„„„ded,^ .he following appended'eUims Ti:^^ L 
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including all such alterations, permutations, and equivalents as fall within the tnie spirit and scope of 
the present invention. 
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C L A r M 5; 



J A method for providing force feedback over a network comprising: 
establishing a connection between a server machine and a client machine over a network said 
.0 fellrk " ' ' humar^computer .nterface device havtng 'fot.e 

feedba TT'"' ^'^^^^^ ^'^P'^'^ and force 

feedback informatton related to said screen display infonnation; 

displaying on said visual display o^'said client machine a sc.en .mage based upon said 
screen display infonmation; 

" screen imre''"' ^ "^'""^ "'"^ '''' '"'"'"''""'P"'^^ ^"^'^^'^^ ^^^^^ -''^ respect to said 
computing a force feedback signal based upon said pointer event; and 

providing a force feedback with said human/computer interface device based upon said foice- 
feedback signal that is correlated to said screen image on said visual display. 



20 



2. 



A method for providing force feedback over a network as recited in claim 1 wi^^^in 
said server machine is a web sender, said client machine is a network access computer. 

3 . A method for providing force feedback over a network as recited in claim 2 wherein 
-5 said web server and said network access computer communicate over said network using TCP/IP 
protocols. ^ 



4. A method forproviding force feedback over a network as recited in claim 2 wherein 
said pointer event is at least one of a pointer icon movement on said visual display with respect to 
30 said screen image and a button click of said human/computer interface device 



5 . A method for providing force feedback over a network as tecited in claim 3 wherein 
said human/computer interface device has at least two degrees of freedom. 
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1 0 sa,d human/computer interface has exactly two degree, of freedom. 



8. 
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A method for providing force feedback over a network as recited in claim 5 wherein 

nirrrr '"'^'^^^ ' - -^^^^ —cater:; 

saxd n twork access cotnputer. said h.m4n/computer interface .nclud.ng sensors and actuators 
coupled to sa.d local microprocessor, and wherein said force-feedback signal is a f^rce fcZ 
command. s„d method funher compnsing: 'ccooacit 

of .JT'I'" " ---P^-"- » said acu,a,o. 

i- jr™ " '"""^ ' Of huma„,comp„,„ 



and said force-reedback information. 



said HTML fi.e^ T"'"' ^ """^^^ ^^'^""^ ^ wherein 

a d 7 V " ' "^^^ -^^^ ^ f-db-'^ object on 

said client machine, and ftirther comprising: 

parsing on said client machine said HTML file and rn-arJoo ^„ o, ,4 i- 
30 feedback object. ^ ""^^'"^ ^^''^ 



comprising: ^ ^ """^^"^ "^'^'^ ^ farther 

35 andsaidrrdJltZir"^ ^" "^^'^ "'"^^^ ^^^^ ^^'^ — 
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1 2. A method for providing force feedback over a network as recited in claim 10 wherein 
said computing a force feedback signal includes: 

monitoring for a pointer event from said human/computer interface; and 

using said force feedback object and said pointer event to provide said force feedback signal. 



13. A method for providing force feedback over a network as recited in claim 1 wherein 
sa.d client machine is a first client machine, and further compdsing a second diem machine coupled 
to sa,d network, said second client machine including a visual display and a. human/computer 
interface capable of providing a pointer event^,said method further comprising: 

establishing a connection between said first diem machine and said second digm machine 
15 over said network; 

receiving at said first client machine second cliem information from said second cheni 
machine; 

computing a force feedback signal based upon said second client information; and 

providing a force feedback with said human/computer interface device based upon said force' 
20 feedback signal. 



14. A method for providing force feedback over a network as recited in claim 1 3 wherein 
said server machine is a web server, said first client machine is a network access computer, said 



second client machine is a network acces.<; computer, and said web server, said first network access 
computer, and said second network access computer can communicate with said network using 
TCP/IP protocols. 



15. A method for providing force feedback over a network as recited in daim 1 4 wherein 
said pointer event is at least one of a pointer icon movement on said visual display of said first 
30 network access computer wid, respect to said screen image and a button dick of a human/computer 
mterface device of at least one of said first network access computer and said second network access 
computer. 



16. A method for providing force feedback over a network as recited in claim 1 5 wherein 
35 sa,d human/computer mterface devices of said first network access computer and said second 
network access computer each have at least two degrees of freedom. 
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1 7. A method for providing force feedback over a network as recited in claim 1 6 wherein 
said human/computer interface devices each have exactly two degrees of freedom. 



: 8 . A method for providing force feedback over a network as incited in claim 1 7 wherein 
10 said human/computer interface devices each include a voice coil actuator for providing at least a 
ponion of said force feedback. 
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19. A method for providing force feedback over a network comprising: 

establishing a connection between a first computer and a second computer over a network, 
where said first computer includes a computer input device providing a first computer input, and 
wherein said second computer includes a visual display and a human/computer interface device 
providing force-feedback in response to a force feedback signal and providing a second computer 
input: 

displaying on said monitor of said second computer a screen image that is associated with 
stored force feedback information; 

receiving from said first computer over said netw.,rk a first computer input associated with 
said screen image on said second computer; 

determining said force feedback signal based upon said stored force feedback information, 
and at least one of said first computer input and said second computer input; and 

providing a force feedback with said human/computer interface device in response to said 
force feedback signal that is correlated to said screen image on said second computer. 



20. A method for providing force feedback over a network as recited in claim 1 9 wherein 
30 said first computer and said second computer communicate over said network using TCP/IP 
protocols. 



21. A method for providing force feedback over a network as recited in claim 1 9 wherein 
establishing a connection includes the exchange of URLs between said first computer and said 
35 second computer. 
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5 22. A method for providing force feedback over a network as recited in claim i9 wherein 

.d f,r. computer input and sa.d second compter input compose a pointer event mCudmg at I 
one of a pointer movement and a button click. 



10 comprising: ^ "''^^ ^ ^"P^""*"^ ^^^^^^ P-ocols 

sending from a client computer over a network supporting TCP/IP protocols a connection 
request to a web server connected to .said network that is hosting a desired URL; 

recetving and parsing an HTML file at said client computer that was sent from said web 

orce object reference havmg associated parameters and building a force object as specified by said 
force object reference and said associated parameters; ■ . ea oy said 

developing a force feedback signal with said force object; and 

providing force feedback ,o a human/computer interface device coupled to said client 
computer in response to said force feedback signal. 



20 



25 



24 A method for providing force feedback over a network supponing TCP/IP protocols 
asrecued,„claim23whereinsa.dKrMLfi,ecomprisesawebpageofsaLwebLer. 

25. A method for providing force feedback over a network supporting TCP/IF protocols 
as recrted in claim 24 wherein a network browser is provided on said client computer to perform said 
parsing and developing said force feedback signal. 



26 

as recii 

30 



as recited , ^ ^ ^"PP°"i"g "^C^^P P-'-ols 



as rec , H f "t. ' """^^'^ ^"PP°"i"g TCP/IP protocols 

as recited in claam 26 wherein said human/computer interface device is provided with a local 
mtcroprocessor that communicates with said client computer, and wherein said force feedback signal 
■s a force feedback command sent from said client computer to said human/computer interface device 
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28. A method for providing force feedback over a network supporting TCP/IP protocols 
as recited in claim 27 wherein said local microprocessor can interpret JAVA instructions sent from 
said client computer. 



29. A method for providing force feedback over a network supponing TCP/IP protocols 
as recited in claim 27 wherein said human/computer interface device includes actuators and sensors, 
and funher comprising: 

parsing on said local microprocessor said force feedback command to control said actuators 
in a feedback loop with said sensors. f 
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30. A networked force feedback system comprising: 
a network; 

a first computer coupled to said network: and 



a second computer coupled to said network, said second computer including a visual display 
and a human/computer interface device capable of providing a second computer input and capable of 
providing force feedback in response to a force feedback signal provide^ by oaid second computer, 
said second computer developing an image on said visual display that is associated with stored 
feedback information, wherein said second computer produces said force feedback signal in response 
to at least one of infomiation derived from said first computer and of said second computer input. 



31. A networked force feedback system as recited in claim 30 wherein said first computer 
is a server computer, said second computer is a network access computer, and wherein said 
information denved from said first computer includes screen display information and force feedback 
information related to said screen display information. 



32. A networked force feedback system as recited in claim 31 wherein said information 
derived from said server computer is sent from the server computer to the network access computer 
in the form of an HTML file. 



^ networked force feedback system as recited in claim 3 1 wherein said information 
derived from said server computer is .sent from the server computer to the network access computer 
in the form of a VRML file. 
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34. A networked fon:e feedback system as recked in claim 30 wherein both sa,d first 
computer and said second computer are network access computers which communicate over said 
network usmg TCP/IP protocols. 



35. A networked force feedback system as recited in claim 34 wherein said infomiation 
denved from sa,d first computer includes infonnation that affects the force feedback signal. 

36. A networked force feedback /ystem as recited in claim 35 wherein said information 
denved from said first computer and said second computer input both comprise at least one of a 
pomter icon position input and a button click input. 



20 
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30 



37. A networked force feedback system as recited in claim 30 wherein said 
human/computer interface device coupled to said second computer includes a local microprocessor 
that communicates with said second computer, a plurality of actuators for providing a force feedback - 
and a plurality of sensors for sensing positions of said human/computer interface device 



38. A networked force feedback system as recited in claim 37 wherein said 
human/computer interface device coupled to said second computer includes a base for receiving input 
from said user, said base being movable in two linear degrees of freedom 



39. A networked force feedback system as recited in claim 38 wherein said base is 
receptive to a fmger of said user for manipulating said base in said two linear degrees of freedom. 

40. A networked force feedback system as recited in claim 38 wherein said base is 
receptive to a stylus for manipulating said base in said two linear degrees of freedom. 



41. A networked force feedback system as recited in claim 37 wherein said force feedback 
signal includes a force feedback command that can be parsed by said local microprocessor such that 
said microprocessor can control said actuators in response to said force feedback command in a 
35 control loop with said sensors. 
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42. A networked force feedback system comprising: 
network means; 

first computer means coupled to said network means: and 

second computer means coupled to said network means, said second computer means 
mc ad.ng visual display and human/computer interface n.eans. said second computer means funher 
mclud>ng means for displaying an image on said visual display that is associated with stored force 
feedback information of said second computer means, said second computer means further including 
means for anaiyzmg infom,ation sent to said second computer means from said first computer means 
over sa.d network means, said second computer means funher including means for developing a 
force feedback at said human/computer interface means using said stored force feedback infomiation 
■n response to at least one of said information sent by said first, computer means and a second 
computer mput providing by said human/computer interface means. 



43. A networked force feedback system as recited in claim 42 whe^in said first computer 
means .s a server computer means, saad second computer means is a network access computer 
means, and wherein said information derived from said first computer means includes screen display- 
mformaf on and force feedback information related to said screen displav information 



44. A networked force feedback .system as recited in claim 43 wherein said information 
denved from sa.d server computer means is sent from the server computer means to the network 
access computer means in the form of an HTML file. 



30 



45. A networked force feedback system as recited in claim 42 wherein both said first 
computer means and said second computer means are network access computer means which 
communicate over said network using TCP/IP protocols. 

deriv tT'''^"' ^'^"^ information 

denved from sa.d first computer means includes infonriation that affects said force feedback signal. 

35 d. ^ -^^-o:k.d force feedback system as n^cited in claim 46 wherein said information 

denved from sa.d first computer means and said second computer means input both comprise at least 
one ot a potnter icon position input and a button click input. 
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48. A networked force feedback sy.<;,em as recited in claim 42 wherein said 
human/computer means interface device coupled to said second computer means includes a local 
m,croprocessor means that communicates with said second computer means, a plurality of actuator 
means for providing a force feedback, and a plurality of sensor means for sensing poshions of said 
numan/computer means interface device. 



49. A networked force feedback system as recited in claim 48 wherein sa.d force feedback 
stgnal mcludes a force feedback command that can be parsed by said local microprocessor means 
such that sa,d microprocessor means can control said actuator means in response to said force 
feedback command in a control loop with said^sensor means. 
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